INTRODUCTION TO DATA

Tags: #◇Computer Science #◇CSMODEL #◇Conceptual #◇Introduction to Data

◇Overview

Data Science is an interdisciplinary field that fundamentally bridges the gap between raw data and actionable knowledge. By synthesizing computational techniques, statistical analysis, and deep domain expertise, it allows practitioners to extract meaningful patterns from increasingly vast and complex datasets. The typical data science pipeline forms the backbone of any data-driven project, systematically guiding the process from the initial data collection and structural cleaning phases, through exploratory data analysis (EDA), to the final stages of mathematical modeling and stakeholder communication.

◇Key Definitions

• Data Science: The scientific domain combining IT/Computer Science, Mathematics/Statistics, and Business/Domain Knowledge to draw insights from data.
• Population: The exhaustive set of all individuals, objects, or events of interest in a study ($N$).
• Sample: A representative subset carefully selected from the population to allow for inferential analysis ($n$).
• Parameter: A fixed numerical characteristic of an entire population (e.g., $\mu, \sigma$).
• Statistic: A numerical measure calculated from sample data, used to estimate population parameters (e.g., $\bar{x}, s$).
• Observational Study: Research where characteristics are merely observed and recorded, identifying correlations but never establishing causation due to potential confounding variables.
• Randomized Controlled Trial (RCT): An experimental design involving the intentional application of treatments with random assignment, uniquely capable of establishing causal relationships.

◇Theoretical Framework / Core Concepts

Types of Data by Structure

Modern data architectures can be categorized based on their schema strictness:

1. Structured Data: Highly organized and tabular, natively conforming to relational schemas. This data is intuitively queried using SQL and stored in relational database management systems.
2. Semi-Structured Data: Lacks a strict tabular schema but utilizes tags and hierarchical markers to structure semantics. Common formats include JSON, XML, and document-oriented databases like MongoDB.
3. Unstructured Data: Raw formats devoid of any pre-defined structural models, such as images, natural text, and audio. It necessitates specialized processing like natural language processing (NLP) or computer vision for feature extraction.

Types of Variables and Levels of Measurement

Variables are the fundamental units of observation and can be split into Qualitative (Categorical) and Quantitative (Numerical) branches. The classification of these variables rigorously dictates the mathematical and statistical operations that can validly be applied to them. This is formalized in the NOIR Taxonomy (Nominal, Ordinal, Interval, Ratio).

The NOIR Taxonomy

Sampling Methods

Sampling fundamentally dictates the external validity of statistical conclusions.

• Simple Random Sampling (SRS): Every possible subset has an equal probability of selection.
• Systematic Sampling: Elements are chosen at regular intervals ($k$) from an ordered frame.
• Stratified Sampling: The population is divided into homogeneous strata, and an SRS is drawn from every stratum. This ensures representation across key subgroups.
• Cluster Sampling: The population is divided into heterogeneous clusters. Entire clusters are randomly selected, and all elements within those chosen clusters are sampled.

◇Worked Examples / Protocol Walkthroughs / Calculations

Example 1: Calculating the Systematic Sampling Interval

Imagine a quality assurance scenario where an engineer must sample 50 microchips from a batch of 2,000 continuous units.

1. Define the population size $N = 2000$.
2. Define the target sample size $n = 50$.
3. Calculate the interval $k = \lfloor N / n \rfloor = 2000 / 50 = 40$.
4. Randomly pick a starting index $r \in [1, 40]$. Suppose $r = 12$.
5. The selected indices are $12, 52, 92, 132, \dots$ until 50 chips are collected.

Example 2: Python Data Type Verification

Using the pandas library, one can programmatically inspect the structure and data types of a dataset.

import pandas as pd

# Load dataset
df = pd.read_csv('housing_data.csv')

# Inspect dimensions and data types
rows, cols = df.shape
print(f"Dataset has {rows} rows and {cols} columns.")
print(df.dtypes)

# Summarize central tendencies and dispersion
print(df.describe(include='all'))

◇Summary & Key Takeaways

• Data structure dictates processing complexity, moving from simple Structured data to complex Unstructured forms.
• The NOIR taxonomy (Nominal, Ordinal, Interval, Ratio) is crucial for selecting appropriate statistical methods; you cannot validly compute an arithmetic mean on Ordinal or Nominal data.
• While observational studies can only suggest associations, meticulously designed randomized experiments are required to prove causality.
• Proper sampling techniques (like Stratified and Systematic sampling) mitigate bias and ensure sample statistics accurately estimate population parameters.

◇References & Navigation

• Next Topic: ◇Data Collection
• Related Notes: ◇Data Collection, ◇Fundamentals of Data and Information